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PREFACE 



The experiences described in this paper occurred in an 
experimental-demonstration project, funded by the Departments 
of Health, Education, and Welfare and Labor under the Manpower 
Development and Training Act. The program is conducted at Draper 
Correctional Center, a state prison in Alabama. 

The Draper Vocational E&D Project has been in operation 
since September of 1964 and serves an incarcerated youthful 
offender population. The project* s purpose is to provide a 
special program for the selection, counseling, testing, assess- 
ment, training, placement, and follow-up of inmates whose many 
problems prevent their profiting from conventional programs in 
vocational training. Programmed instruction and several allied 
training methods are being developed and used to instruct the in- 
mates in an effort to overcome their defeatist attitudes and to 
reduce the vocational training time without sacrifice of the 
quality or quantity of their learning. 

In order to make its findings of value to other prison 
systems and similar training programs for the disadvantaged, 
the Draper project is currently preparing guidelines for dis- 
semination and utilization. While the Federal Government spon- 
sors encourage E&D Projects to express their own judgment freely, 
the points of view stated in this report do not necessarily rep- 
resent the official position or policy of the U. S. Departments 
of H.E.W. or Labor. 



DEVELOPMENT, EVALUATION, AND USE OF PROGRAMMED MATERIALS AS DEVELOPED 
IN THE DRAPER EXPERIMENTAL AND DEMONSTRATION PROJECT • 

Donna M. Seay 

Rehabilitation Research Foundation 

Elmore, Alabama 36025 

A number of years ago I became interested in programmed instruction 
(P.I.)--a new instructional method which seemed to be ideal for vocational 
education. P. I. had certain characteristics that allowed an instructor 
to individualize his course according to the needs of each student. More 
impressive than the characteristics were the results obtained with this 
method. Learning was assured, even though the rate of learning varied with 
the individual's ability and interest. Programmed instruction permitted the 
instructor to devote more time to the students requiring special attention. 

At this time, I was a frustrated Distributive Education Coordinator 

t 

looking for instructional materials which would make my course more effective. 
Here I was, trying my best to teach at least 20 different subjects at the 
same time since each of my 20 students was placed in a different distribu- 
tive occupation. Of course, I had study guides for related information in 
each subject, but their use presented a number of problems. For instance, 
adaptation of the material was almost always required. Checking answers 
to questions on each job sheet was an endless task, not to mention the job 
of testing every student: as he completed a lesson. 

When I first heard of programmed instruction and its advantages, I 
began investigating the possibility of using these materials in my related 
study classroom. I could imagine myself assigning each student a programmed 
course with the assurance that he would learn whatever was required for his 
occupational training. Unfortunately, such was not the case. My 

Presented at the Regional Manpower Development and Training Conference, 
Region IV., U.S. Department of Health, Education, and Welfare, Bureau of 
Adult and Vocational Education, Atlanta, Georgia, March 8, 1967. 



investigation did not turn up the kinds of programs which would have been 
appropriate for Distributive Education (DE). However, I learned enough 
through reading research reports on P. I. to become convinced that its use 
was an improvement over the study guide and other traditional aids to 
instruction. 

Since DE programs were not on the market at the time, I began to 
study the different techniques used in the development of P. I. materials 

in the hopes that I could learn to write them. The more I studied, the 

* 

more I realized that programming is no easy task, particularly if one is 
going to write programs which will truly teach. 

Fortunately, I heard about a unique educational experiment in the 
use of P. I. materials with inmates at Draper Correctional Center, Elmore, 
Alabama. This experiment was conducted by Dr. John M. McKee, a clinical 
psychologist, who was at chat time the State Director of Mental Hygiene. 
Later, he resigned his position with the state to accept the full-time 
job of Director, Draper Experimental and Demonstration Project in Academic 
Education, which was financed by the National Institute of Mental Health. 
(This project has been in operation for the past five years.) One of 
the findings of this experiment pointed up the need for additional 
education in the area of vocational training. 

Consequently, Dr. McKee asked Mr. J. F. Ingram, Director, State Divi- 
sion of Vocational Education, for advice and assistance in planning a 
vocational training program. After several discussions with Mr. Ingram 
and other MDTA officials, Dr. McKee decided to submit a proposal for an E&D 



Project under the MDTA. As the result of Mr. Ingrains reference to 
me as a vocational educator who was interested in P. I., Dr. McKee 
asked me to serve as a consultant in planning and writing the proposal 
which was approved in September, 1964. 

Needless to say, my original interest in P. I. and my involvement in 
developing the proposal for the MDT Project led me to give up my job as 
a DE Coordinator to become the MDT Program Director under Dr. McKee's 
direction as administrator of the contracting agency, the Rehabilitation 
Research Foundation. The Foundation is a private, non-profit organization 
that is presently conducting research in human behavior. 

One experimental and demonstration feature of the vocational training 
project is the Materials Development Unit (MDU) which is responsible for 
investigating and developing programmed instructional materials. For 
example, the investigation includes the evaluation and use of programs, 
and the methodology of programming. The unit develops all types of programs 
and other special training materials, such as wall charts, diagrams, and 
transparencies for overhead projectors. 

Presently, the MDU staff consists of an editor -coordinator , one 
program writer, one artist, two production assistants and subject-matter 
specialists who work by the hour when needed. Each member of the staff 
performs several tasks with respect to instructional materials. However, 
since my topic pertains to our programmed materials^ X shall confine my 
remarks to the step-by-step procedures necessary in the development, 
evaluation, and use of the individualized lessons that I have on display 
here today. These mathetieal training materials are different from 



traditional training publications in that they reflect the principles 
and techniques defined and developed by several experimenters who have 
systematically applied the reinforcement learning theory of B, F. Skinner • 

The system of mathetic.s, which was (developed by Thomas E. Gilbert, 
is used by our MDU staff in preparing the programmed lessons. Gilbert 
defines mathetics as. 0 . ,! the systematic application of reinforcement 
theory to the analysis and construction of those complex behavior reper- 
tories usually known as ’subject-matter mastery, 5 ’knowledge,, 5 and 

X 

'skill. 5,1 (It should be pointed out that, as programming has come 
of age, there appear to be more similarities than differences in the 
various programming techniques.) 

The gc 1 of every matheticist, an analyst -writer of mathetical lessons, 
is to work toward a genuine technology of education by combining in his 
programs the concepts of behavioral science with the effective practices 
and procedures that have always been used by good teachers. 

Perhaps the easiest way to understand how these behavioral science 
concepts may be combined with effective instructional practices and procedures 
is to describe the mathetical system which our unit uses in developing 
programmed lessons. 

Practices and Procedures of the MDU 

The MDU uses an exacting and systematic process to develop and to 
improve existing materials so that they are student -oriented and student - 
proved . 

^Gilbert, Thomas E., ,v Mathetic&: The. Technology of Education, " Journal 

of Mathetics , Vol. 1, No. 1, January, 1962, p. 8. 
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The mathetical approach involved functions requiring participation cf 
specialists as well as staff. Skilled technicians in each vocational area 
decide what subjects should be programmed. They also serve as subject- 
matter specialists by choosing for the writers the appropriate practices 
and procedures within the selected areas* Our vocational instructors act 
as specialists or experts. In addition, we usually ask other technicians 
or professionals in the same vocational area to verify the content of the 
training lesson. 

Recently, a committee made up of vocational educators in Alabama met 
together to discuss the topics we should program this year. After much 
discussion the group selected "Communication Skills for the Auto Mechanic." 
as an area where training materials are much needed. The staff of the 
MDU is now in the process of reviewing literature and interviewing 
the experts in this field. 

In order to produce programs, the MDU performs certain functions 
which fall under the following general headings: 

1. Subject Matter Selection 

2. Specification of Operational Deficiency 

3. Performance Requirements 

4. Performance Analysis and Programming 

5. Editing and Evaluation 

Forgive me if I use technical terminology or fail to explain fully as 
I discuss these functions. The subject is a complicated one, and time is 
short. After all, :!t takes approximately six months to train a programmer! 
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1. Subject-Matter Selection 

The first function, subject-matter selection, proved to be very 

tedious, time consuming, and costly in the initial stages. However, 

* 

when we failed to give this function adequate consideration, we ended 
up with some lessons that did not fit into every instructors course 
outline. When subject matter is properly selected, the cost of the 
programming is justifiable in terms of the learning time saved, and 
the programs have high standards and broad application to training. 

To make sure that subject matter is properly selected, it is necessary 
to first determine the extent to which a particular performance 
deficiency is a widespread and significant problem. In other words, 
there should be a large audience with a real need for the program. 

As a rule of thumb, we say that if over 50% of the target population 
knows over 50% of the material the area does not require programming. 

The area selected should also be one that presents teaching or learning 
difficulties. In short, programs are not written to replace existing 
materials which already do an adequate job. They are written if materials 
are non-existent, or if what is available does not teach well, or to 
supplement-- to make teaching and learning easier and more effective. 

There are areas which are better taught by other methods, such as 
demonstrations or group discussion. All I am saying is that there must 
be a valid reason for developing a program--we do not program in a vacuum. 
2 . Specjf icati on_of _0pera tional Def ic iencv - (What do we need to teach?) 

Since the only justification for a program is that it can correct an 
operational deficiency, the training needs and standards of effectiveness 
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are determined on the operational level. If the student does not know how 
to do something, or if he is not doing something correctly, the writer 
states these deficiencies so clearly that there is no doubt about the 
extent to which they can be overcome by subsequent training in the form 
of a program. The formula for assessing knowledge or skill deficiency 
is M - I - D. M is the master's or expert's performance; I is the 
initiate's or trainee's performance; D is the deficiency, the difference 
in the performance of the expert and the novice. 

Once the operational deficiency is determined, it is translated into 
terms of tentative training objectives. These objectives form the guidelines 
for writing a detailed description of the subject matter practices and 
procedures. The analysis of the subject matter and the format design 
of the program are based on the objectives, too. 

The training objectives are stated in behavioral terms--concise, 
measureable terms of what the trainee should be able to d<* after completing 
the program. Such ambiguous terms as "to understand," "to know," or "to 
appreciate" are avoided. Instead, specific behaviors are listed, such as 
"to write," "to identify," "to solve," or "to list." 

For example, "Whan the student has completed the program, he should 
be able to mix mortar"... 

or 

"to identify electrical circuit symbols used by an industrial electrician"... 

or 

"to use a scale ruler"... 

The objectives also state the conditions under which trainees are 
expected to perform after taking the programmed lesson. 
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For example: "Given the necessary materials this student will be able 

to mix mortar to be used in laying a brick wall". . .Negative or delimiting 
requirements would be included also, that is, "This lesson does not teach 
how to estimate the amount of mortar needed." 

Finally, the objectives specify criteria of acceptable post-program 
performance of the trainee, that is, the level of competence at which the 
student should be able to perform. These criteria are usually expressed 
in terms of time, percentage of correct answers on an examination, or actual 
demonstration of ability before a supervisor or examiner. 

Training objectives are. prepared with the prospective trainee population 
in mind. Most of our materials are designed for the disadvantaged trainee; 

however, the programs proved to be even more successful with other groups 

♦ 

who were not necessarily deprived or handicapped. Regardless of our success, 
we always describe the design population in terms of educational levels and 
general background and knowledge in the areas to be covered by the program. 
Since it is not always possible to uncover individual deficiencies of the 
target population before a program is developed, it is sometimes necessary 
to develop remedial programs which will provide the prerequisite knowledge 
. needed to complete a particular program. For example, our fractions labora- 
tory is being developed because the bricklayer trainees were unable to solve 
problems requ;ir ing .the use of fractions in a series of lessons onrestimating 
materials. Individual and field tryouts quickly uncover the remedial areas 
that need to be programmed. 

3. Performance Requirements 

Once the operational deficiency is determined (stated as training 
objectives) and the feasibility of a program is confirmed, the correct 
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performance (reflecting current, standard subject-matter practices and 
procedures) is determined,, This is the "job analysis," Its importance 
can hardly be overstated, for a program can be no better than the. analysis 
upon which it is based. Correct performance is determined by observing 
the actions of an expert practitioner and by questioning him about his covert 
actions, since covert performance is just, as significant as overt. (The 
Material Development Unit°s personnel had. to be trained to ferret out 
obscure behavior.. ) To ensure accuracy, someone considered to be even 
more "knowledgeable" than the practitioner checks the analysis to see 
that the behaviors described are actually those behaviors the trainee should 
learn. As you would perhaps guess, the subject matter experts sometimes dis- 
agree as to what procedures or practices are correct l In such cases, the 
writers consult with other experts, and they also refer to the most up-to- 
date reference materials available. The procedures or practices used are 
those on which most of the experts are in agreement. 



4 . Performance Analysis and Programm i ng 

Actually, this function and the previous one (performance Requirements) 
overlap considerably. The first phase of this function consists of delimiting 
and organizing into behavioral terms the content of the course or program. 

This step is very important, because it defines the initial deficit in the 
capability of prospective, students with respect to subject-matter competency. 
The analysis also helps to determine the maximum "operant span" or step- size 
by which, the student can effectively learn— that is, how much can be absorbed 
at one time. The notational system used in this initial analysis is called 
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a "prescription." In the prescription. the subject matter for the program 
is first broken down into statements of what the trainee is to learn to 
do (the response, or R) and when he is to do it (the stimulus, or S) . 

This technique reveals discrepancies which may be found in standard job 
analyses, and it highlights the overall behavior patterns. 

After the "prescription" is completed, a final check is made for 
technical accuracy. It is possible at this time to determine what the 
program in its final form will accomplish, that is, final training 
objectives are formulated. 

The second phase of the Performance Analysis and Programming includes 
a systematic analysis of the "prescribed" behavior deficit for those 
generalization and competition components that cause the primary learning 
problems for the student. This analysis anwers such questions as: 

1. Are there similar stimuli which may not appear similar to the student 
but which require the same response? For example, having learned the 
sound of "B," will the student know that "b" and "£r' have the same 
sound? If your answer is n No," the generalization must be made for 
him. 

2. Are there stimuli in the prescription which may appear similar to 
the student but which require different responses? For example, 
the scales on the Volt-Ohm-Milliammeter (VOM) almost always appear 
as concentric arcs which are read with the same pointer. Yet, the 
ohms scale is read from right to left while the scales for volts 
and amperes are read from left to right. Teaching strategies must 
be devised that will treat for such competition. 



10 



3 . 



Is there a similar stimulus situation outside the specific behavior 
being taught, but in the student's experiences which may be confusing? 
Most household light switches are installed so that one. turns the 
light on by moving the switch up. Suppose that a machine which the 
student is learning to operate is turned on by moving the switch 
down. He may attempt to turn the machine on in the same way he has 
learned to turn a light on. Again s competition must be overcome. 

The second phase also includes the development of outlines or "lesson 
plans" which show the precise teaching strategies that will be used to 
produce the actual "exercises"- “the term used to describe a teaching unit 
in a mathetical program. 

The teaching strategies used iu these exercises are characteristic of 
mathetical lessons. There is a great deal of flexibility in the layout and 
response requirements since ma the tics is not a format system. Function 
determines the format. Notice the lack of uniformity of style or appearance 
from lesson to lessen or page to page. An exercise uses whatever is best 
depending on the characteristics of the behavior to be taught and the 
abilities of the student population. Some exercises look much like a 
linear frame while some may resemble a double page spread with all the 
design appeal of a good magazine advertisement . 

All types of responses are called for in mathetical lessons. They 
vary from a paper and pencil type response to those involving the use 
of tools or simulator kits. The response is not always overt. 

Because the learning situation should duplicate an actual situation 
as nearly as possible, extensive use of illustrations and simulations 









characterizes mathetical lessons. We find that it is effective to 
represent a particular stimulus by using illustrations to teach the 
student the correct response. Illustrations and simulators assist the 
student in transferring his knowledge from the learning situation to 
the job. Our program, "Soldering Leads 9 " is a lesson in which illus- 
trations and simulations were used very effectively in a program. Boys 
were able to transfer their knowledge very easily without any help from 
an instructor. 

In most cases, a lesser degree of simulation will work well. For 
example, our series on using the VOM actually has a drawing of the 
instrument to guide a student in its proper usd. By marking on a 
drawing at key points or in a certain sequence a student is able to 
apply the knowledge -.0 actual job performance. 

The model teaching exercise presents a stimulus-response relation- 
ship at least three times: once in a "demonstration," then in a "prompt," 

when the student responds with assistance, and finally in a "release," 
when he responds without help of any cues. Students like these lessons 
because they are able, to learn without being bored to distraction by 
repetition that they dislike intensely. 

5 . Editing and Evaluatio n 

The first phase of this function consists of editing procedures that 
are generally standard; however, there are some exceptions which I shall 
explain. 

First-draft: exercises are submitted for review to the subject-matter 
specialist who checks the technical accuracy of each program. Any suggested 
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changes are usually limited to minor points such as technical terminology, 
and do not include changes entailing extensive reanalysis and rewriting. 
After these changes are made, an individual tryout is conducted with a 
student in the design population. In tryouts and field tests, a pretest 
on information covered by the lesson is administered. The student (or 
students) then takes the lesson. A posttest is then administered. The 
difference in pre- and posttest scores tells us how well the lesson 
taught. A student’s failure on certain parts of a lesson may point up 
needed changes. Samples of such changes may be decreasing step size, 
changing layout to eliminate confusion, rewording, etc. If changes are 
made after individual tryout, the program is resubmitted to the subject 
matter persons for review. The most critical phase is the evaluation 
which is based on the individual, and field tryouts. In the individual 
tryout a student takes the program under the close observation of a 
staff member of the Unit. Depending upon the heterogeneity of the 
prospective design population, from one to six such tryouts are conducted; 
one tryout may suffice for a highly homogeneous population. Revisions 
are made to correct inadequacies in the program. The cycle of tryout 
revision- tryout continues until the student's performance reached an 
acceptable level which is normally 85% or above on the posttest. It 
is this tryout procedure we refer to when we say that our programs are 
student - prove d . 

Finally, the program is submitted to field testing, meaning that 
a representative sample from the prospective training population uses 
the program under operational conditions as close as possible to 
conditions of actual performance. (The results of our field tryouts 
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are included in the specifications of each program when it is published. 
A report of the field test results is included in the Programmed Lessons 
brochure which may be obtained from the Rehabilitation Research Founda- 
tion, P. 0. Box 1107, Elmore, Alabama. The back of each lesson cover 
in the brochure gives the specifications for the lesson.) 

Use of the Programs 

Although it was impractical to fit the programmed lessons to the 
curricular schedule of the various field-test classes, it was possible 
to install the lessons in the precise place for which they were designed 
in the curriculum of the courses at Draper. The following data give an 
exact picture of some of the. lessons used in the appropriate place in 
the training schedule. 



Results of 


Programs used 


in Draper * s 


Courses 


Lesson 


Pretest 


Posttest 


Net Gain 


Mixing Mortar 


28% 


97% 


69% 


Tools & Areas of 
a Haircut 


35% 


98% 


63% 



Most important of all Ls the fact that these lessons tend to motivate 
the trainee to continue working. Trainees and instructors are definitely • 
in favor of using programmed materials whenever they are available. 

The flexibility of these mathetical lessons makes them ideal for 
training needs of vocational schools and industry where transfer of 
skills to actual job performance is critical. Because of their flexi- 
bility, their value is not limited to individualized instruction. We 
plan to use the mathetical system in programming group instruction, 
which could be presented through films, slides, role-playing, or other 
techniques. 



o 
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One of the most frequent criticisms leveled at programmed instruction 
particularly mathetical lessons, is that it is expensive to produce. 
Admittedly, the cost of production is greater than that of traditional 
training materials. We believe that the advantages of programmed material 
far outweigh the costs. I will not list the advantages again, but will 
summarize them all by stating that we have demonstrated that programmed 
instruction provides both learning and instructional efficiency. Thus, 
we have achieved one of our major aims. If you would like to improve 
your training program, we highly recommend that your instructors be 
trained to use programs properly. 




Selection and Evaluation of Programmed Instructional Materials^ - 

Martha Terry 

Draper Correctional Center 
Elmore, Alabama 36025 

Whether one is evaluating programmed instruction (P.I.) to determine if 
it is the most efficient method for the teaching job at hand or to determine 
which of two or more programs best meets his needs, the method of evaluation 
is much the same. Once committed to the use of programmed instruction on 
either a Limited or general basis, one. must seek specific programs to meet 
specific needs. The wealth of P.I. materials available confuses even the vet- 
eran practitioner. Before considering any programmed lessons, the evaluator 
must know the answer?. t<* these questions: Are you going to use programmed 

materials to teach an entire course.? Do you plan to use them to supplement 
or enrich the usual course of study--that is, will they be used to help slow 
learners over difficult spots and to allow fast learners to do further study 
while others master the required subject matter? Are there specific points in 
the curriculum at which ymi plan to use P.I. --points which traditionally 
have been instructional stumbling blocks? Will you use them to up-date and 
sharpen the. skills and knowledge of your staff? In other words, to evaluate 
you should know exactly how you plan to use programmed instruction. 

Behavioral Objectives - -You should also formulate in writing the specific 
training objectives you plan to achieve with programmed instruction. These 
objectives should state in precise, measurable terms, exactly what the learner 
will be. expected to do after he has completed the learning experience. A 



•1'This paper was presented at the Draper Training Conference for 21 visitors 
from Hawaii, Montgomery, Alabama. November 21, 1967. 



statement that: the, student will know how to do long division is not enough. 

A better objective would be; Given 50 problems in long division, the student 



will be able to set up and solve A 5 of them. Preparation of "behavioral 
objectives" is sound teaching practice regardless of the method of instruction. 
Preparation of such objectives is essential when one begins to screen programmed 
materials for possible use. You use them to select P.X. materials which have 
approximately the same behavioral objectives as those you have designated. 

One programmed course may completely satisfy a particular training need while 
other deficiencies may require all or part of several P.X. lessons. 

Sources - Just finding out what, is available in P.I. materials is a major 
tasu. For this purpose, two resources appear to be Indispensable : The. 

Automated Education. Handbook and the Hendershot Catalog . Both references 
list: programs by subject matter. Each also contains information about grade 
level 9 price, and publisher. The Automated Education Handbook gives a better 
description of: i:he materials available. In addition, it contains essays and 
discussions on the theory and use of programmed instruction. The Handbook 
is, however, an expensive volume; the Hendershot may suffice strictly for 
ordering purposes. To acquire programs to evaluate, one peruses these 
volumes and orders examination copies of programs which appear to cover the 
desired subject: matter at the. appropriate level and for a price which is 
within his means. 



Now to evaluation. 

Evaluati on » The obvious way to evaluate a program is to try it out on 
the students with whom you plan to use it. Such tryouts may not always be 
possible; furthermore, yuu will not. want to try out every program you receive. 
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For example, certain programs may prove to be totally inappropriate to your 
training needs. Your own training objectives will serve as a preliminary 
screening device . All good programs have definite, objectives and they 
should be stated in the same terms as your own --in terms of performance --what 
can. the student do upon completion of the. program. X.f: your objectives and 
those of the program seem to coincides further examination is called for. 
Programs can be eliminated immediately if their objectives are not: appropriate 
to your needs. As Susan Mark'le says , !k A program takes shape with the specific 
audience in mind** 1 Information ah ut this target audience should be included 
with the readier 1 s guide or manual which should accompany a program. If this 
audience and your own students are somewhat similar, you are ready to subject 
the program to a searching., critical inspection. This inspection should be 
guided by the following key factors? Content, Construction, Level, and 
Pedagogy. These areas are discussed here separately for ease of presentation; 
you wilL not: have to review a program four times. This writer believes that 
a reviewer can best accomplish his task by going through a program as a 
student . 

Content - As you consider content, you will be seeking to answer several 
questions, keeping in mind, your intended use of programmed instruction. First, 
does the program cover the topics you plan to teach a Titles can be misleading. 
1 have on my desk a program, entitled “Effective Writing.” Promotional, material 
led me to believe, it would help me to write "clear, forceful prose." The 
program turned out to be a course in. English grammar. Granted that a grasp 
of grammar, punctuation, and usage is essential to the production of good 
prose, I didn't anticipate that the "secrets of composition" would be quite 
so basic. 
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Second 5 what skills does the program develop? And, are these the skills 

you want your students to develop? For example , a program may teach a student 

to quote, rules tor the use of: the comma. Is this what you want your students 

to be able, to do, or do you want them to be able to use. commas correctly? 

« 

Here. again, we see the importance of: written objectives. 

If you plan to teach a whole course with a program*, you will want to 
see if it: is in line with the prescribed course of study. And regardless 
of use , you will want to know :t.£ what: It, teaches is in agreement with what 
other ’'authorities' 1 ' in. the field, teach 0 If: you aren't: sure, get the opinion 
of: an expert, and, check the author's qualifications and these, of any consultants 
he lists as subject-matter experts » In short:, the subject-matter content: must be 
technically sound, and it must be appropriate to your general and specific train- 
ing needs. 

Construction - The second area you will be considering as you examine 
a program is its construction and fabrication ; that is, the way it is put 
together. Format or construction may s if nif leant ly add to or subtract from 
the. effectiveness and applicability of a programmed lesson. For example, 
you will want to determine whether or not: the program is divided into 
distinct units or: segments • This is particularly important if you are plan- 
ning to assign a supplementary role to programmed instruction. If there are 
logical divisions, you can use parts of a program more readily. A particular 
deficiency may be limited in nature and, therefore, only certain portions of 
a programmed course may be required. 
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Another item of importance is the ease with which a program can be 
used. The sheer mechanics of a program can make it frustrating. The 
physical negotiations required (turning pages in an unorthodox manner 9 
inverting the programmed text, etc.) often presuppose more aptitude than 
does the subject matter of the program. The ease with, which a student can 
confirm his responses should also be considered. He should not have to 
expend undue effort to check his answer; yet it shouldn’t be easy for him 
just to copy correct responses. 

Some other questions you will want to answer are. theses Are the 
directions easy to follow? Does the program require special equipment? If 
so a will such special equipment be available to you? Will there be enough 
of it for all of your students? Will this equipment be cumbersome, or 
inconvenient to use? 

You will also want to determine if the program is consumable or reusable. 

It isn’t necessarily desirable to reuse a program; your instructional materials 
budget must dictate. Be aware , however, that the effectiveness of some programs 
would be altered significantly if students are not allowed to respond in the 
program itself. For other programs, having students to write responses on 
notebook paper will not change the effectiveness. To some, extent, you can 
be guided by the publisher’s recommendations here, but rely on your own judg- 
ment , too • 

Level. - Another factor in. evaluating a program is determining its 
difficulty level. ‘This determination must be more, precise than, simply 
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saying, "Course X teaches English grammar at the 7th grade level." Such 
sweeping phrases are common in bibliographic descriptions of P.X. materials. 
Remember, your needs are determined by your "behavioral objectives;" you 
are looking for program context that will teach these specific objectives. 

In this context, the term "grade level" is not too meaningful. What is 
important is the reading level, of the material. In many cases, the reading 
skills called for are at a much higher level than the subject-matter content. 
Is the program written in a style and with a vocabulary your students can 
understand? If there is technical terminology are your students familiar 
with it? Or is it taught by the program? In passing, let me say that it*s 
probably better to err on the side of too low a level than one that is too 
high. 

You must also identify the. prerequisites for each course. A certain 
series of lessons on estimating materials requires the use of fractions in 
a problem-solving context. If the bricklaying trainees for whom this 
series was designed do not have these necessary skills the lesson is of 
no value. In an otherwise sound P.X. lesson, disregard for the prerequisite 
skills may render it totally ineffective. If the target population does not 
have the prerequisites, remedial material must be prescribed. Too many 
deficiencies might lead one to reassess his target population. 

Pedagogy - The. fourth and final area you consider as your inspect a 
program is pedagogy. Actually, what is needed here is in the nature, of a 
warning. I am quoting from an article by Paul I. Jacobs in the Automated 
Handbook : "The way the subject matter is organized and presented in a 

program is likely to surprise, you. The order in which topics are covered may 
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be strikingly different from other presentations of the 'same® subject matter 
that you have seen in textbooks. The steps the. student takes to master a 
given topic, may seem too small or repetitious. Think twice before rejecting 
the program on these grounds alone. It may be just these features that make, 
the program uniquely effective." 

After you have completed your inspection of the programs, you should, 
know which you want to try out, or at least have eliminated those you don't 
want to try. If you still are doubtful, review the research evidence which, is 
furnished with the program. And beware the program which lacks such data. 

This information should, tell you how the program was tried out by the author 
and/or publisher. It should state who the students were, how the program 
was used, conditions of testing, and what results were obtained. That is, 
pre- and posttest scores should, be furnished, along with copies of the tests. 
It should tell you how long the. tryout students took to complete the test. 

The attitude of the students toward the program and the method of ascertaining 
it should be. reported. Even when a purely subjective judgement has been made, 
it may have some validity. 

At last you are ready to try the program or programs which have survived 
your inspection. Administer the. lesson, to a small group and. carefully observe 
their performance as they work. Administer pre- and posttests so that you 
will have a measure of how well the program taught. The results --test scores 
and your own observations --give you a basis for deciding whether or not a 
program can be used for your purposes. 
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Earlier it was suggested that you may not always be able to try a program 
out. For example, suppose you want to teach an entire course with programmed 
materials. The ideal way to determine the program 8 s merit would be to 
administer it to a class, using as a control group a class being taught by 
conventional methods. Your two groups would be administered the same pre- 
and posttests. The results would tell you if your program is at least as 
good as conventional teaching. In a public school system this just isn*t 
practical. You run the risk that the program does not teach as well, and 
you then have a group of students who have been short-changed for a semester 
or a year. This is the place where you must rely on the experience of others. 
It 8 s better not to rely on the judgment of someone who has inspected a 
program and written a review. If at all possible, find out what someone 
who has actually used the program thinks of it. If inspection alone- -others 
as well as your own--is your only criteria, another warning from Paul I. 

Jacobs is in orders "In our present state of knowledge, different inspectors* 
of a program may not agree on its teaching effectiveness, or, even if they 
do agree, they may not be right. If you nevertheless want to or if circum- 
stances compel you to place primary weight in your decision on your inspec- 
tion of the program, then you will find a book by Markle quite helpful." 

The book to which he refers is Susan Markle ®s Good Frames and Bads A Grammar 
of Frame Writing . 

Evaluation involves inspection, review of research evidence, tryout, 
and the opinions of others. Anyone who begins to evaluate programmed 
instruction will doubtless find other questions to ask, and he will doubtless 



discover that not all questions suggested here are apropos to all programs » 
This is an attempt to furnish guidelines based on our experience. The sum 
of that experience is this: 

No programmed instructional material is intrinsically valuable. It 
must be considered in the context of target population , adaptability to 
curriculum , time , motivational characteristics , measurable outcomes, and 
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